SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR SELECTIVE 
REPLACEMENT OF OBJECTIONABLE PROGRAM CONTENT WITH LESS- 
OBJECTIONABLE CONTENT 

COPYRIGHT NOTIFICATION 
5 [0001] Portions of this patent application contain materials that are subject to 

copyright protection. The copyright owner has no objection to the facsimile reproduction 
by anyone of the patent document, or the patent disclosure, as it appears in the Patent 
and Trademark Office. 
BACKGROUND OF THE INVENTION 
.10 1. Field of the Invention 

[0002] The present invention relates, generally, to the field of processing of audio 
and/or video signals, such as for example television broadcasts, movies, and other 
programs, and, more particularly, to a system, method and computer program for 
selective replacement of objectionable content in a program with less-objectionable 
15 content, such as, ocl advertisements. 
2. Description of the Related Art 

[0003] It is a common desire of many viewers or listeners of television, movies, music, 
and other programs to filter portions of the audio and/or video of such programs. For 
example, many parents prohibit their children from watching cable television 
20 broadcasts, certain movies, or other programs because the audio of the program may 
include inappropriate language or the video may include inappropriate content, such as 
nudity, sexual content, or violence, that is unsuitable for children. In addition, since the 
parent has not seen the program, the parent may not know whether the content of the 
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program is inappropriate for the children or not. Consequently, parents may 
inadvertently permit their children to view a program not knowing portions of it are 
inappropriate, or conversely, may prohibit viewing of the program even though its 
content does not include any inappropriate portions. 
5 [0004] In an effort to solve this dilemma, the U.S. Federal Communications 

Commission ("FCC") has established provisions (EIA-608-B) for attaching parental 
guidelines ("ratings") to broadcasts using any of four different systems (U.S. TV, U.S. 
MPAA, Canadian English, and Canadian French). According to the FCC guidelines, 
however, only one rating may be applied to any given program and that rating may not 

1 0 change during the course of the program. 

[0005] The inability to change the rating midway during the broadcast makes the FCC 
parental guideline system a "go or no-go" system. In other words, either the entire 
program is deemed appropriate or inappropriate depending on the rating. 
Consequently, while the rating system is helpful to parents in determining whether to 

15 permit the child to view the entire broadcast or program, it does not facilitate filtering of 
portions of the program to prevent viewers (e.g., children) from seeing or hearing 
particular portions of the program that are objectionable. 

[0006] There are various prior art systems and methods that are based on or rely on 
the use of a government or industry-wide program rating system, such as the previously 
20 discussed rating system, that use a "V-chip" approach, or that use a proprietary rating 
system. See, for example, U.S. Patent No. 5,485,518 to Hunter et al. Such systems 
are effective for filtering audio and/or video content that the user or viewer indicates 
should be blocked based on such content being associated with a specific rating. The 
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specific ratings, however, are applied to an entire program and, as such, the systems 
that block audio and/or video content based on such ratings block or allow through the 
entire program. 

[0007] These systems are "go or no-go" systems, as discussed above. Either the 
entire program is deemed appropriate and let through, or inappropriate and blocked 
depending on the rating. There is no capability in such systems for allowing the user or 
viewer to watch the program, but to replace objectionable material (audio or video) with 
less-objectionable material, such as, ag. advertisements. 

[0008] Furthermore, many of the various prior art systems that provide for filtering of 
audio and/or video content require specialized hardware or additional components to 
perform the replacement function, such as audio-video devices that use dedicated 
circuitry, components, or chip sets. See, e.g., U.S. Patent No. 5,828,402 to Collings; 
U.S. Patent No. 4,554,584 to Elam; and U.S. Patent No. 5,387,942 to Lemelson. While 
these systems are able to block content deemed to be objectionable by the viewer or 
user, the implementation of the filtering function is dependent on inclusion of the 
required hardware into the audio-video device. Such necessary hardware increases the 
cost of such equipment, makes upgrading or updating to improve functionality and/or to 
add features difficult and, in most cases, impossible, and severely limits widespread 
adoption and use. 

[0009] U.S. Patent No. 6,230,320 to Gakumura discloses a system and method for 
temporarily releasing viewing restrictions in a television receiver employing a V-chip 
system. In order for the user to view the blocked program, the user must enter a 
viewing restriction release command to temporarily release viewing restriction based on 



the contents of a rating set. This entry results in the viewing restriction based on the 
contents of the set rating being invalid with the contents of the set rating held when the 
viewing restriction release command is entered. The program can be reblocked by the 
user entering a viewing restriction return command to return the viewing restriction, 
which makes the viewing restriction based on the contents of the set rating valid. While 
this system provides a temporary way to unblock a previously blocked program, it does 
not provide the capability of allowing the user or viewer to watch a program, while 
replacing objectionable material (audio or video) with less-objectionable material, such 
as, exj. advertisements. 

[0010] U.S. Patent No. 6,075,550 to Lapierre, which is hereby incorporated by 
reference, discloses a system that filters the closed caption and/or audio of a television 
program by comparing the words in the transmitted closed caption with a list of 
offensive words. When an offensive word is detected, the system will remove it from 
the closed caption stream so that it is not displayed on the television screen. Based on 
detection of the offensive word in the closed caption stream, the system mutes (i.e., 
turns off) the audio signal for a timed period. 

[0011] The system estimates the time at which it should start muting the audio and 
the length of the timed period. The disclosure indicates that the period of time for which 
the audio signal is turned off is flexible. A longer period during which the signal is 
turned off tends to decrease the chance of objectionable material getting through; 
however, this also tends to filter a greater amount of unobjectionable material. A 
shorter period during which the signal is turned off tends to have the opposite effect. 
[0012] The period of time for which the audio signal is turned off may also be a 



function of the word or phrase encountered. Where a particularly objectionable word, or 
particular lengthy objectionable phrase is encountered, the audio signal may be turned 
off for a relatively long period of time. 

[0013] Similarly, where the word is less objectionable and does not involve a lengthy 
5 phrase, the audio signal may be turned off for a relatively short period of time. Thus, 
the system estimates when in the audio signal the word is likely to appear, based on the 
arrival of the word in the closed caption stream, to determine when, and for how long, to 
mute the audio. While the system is effective in filtering offensive words from the closed 
caption stream, filtering of the audio signal is based on an estimate and may result in 

10 audio that is not objectionable being blocked or audio that is objectionable getting 
through. Plus, the system does not provide the capability of replacing objectionable 
material with less-objectionable material, such as, acu advertisements. 
[0014] Further, the system does not address objectionable content that may be 
present in the video signal. See also U.S. Patent No. 6,166,780 to Bray. 

15 [0015] Thus, notwithstanding the various systems and methods of the prior art, there 
is a need for a system, method, and computer program product that provides for the 
selective replacement of objectionable material (audio or video) with less-objectionable 
material, such as, ag^ advertisements. Further, there is a need for a system, method, 
and computer program product that provides such selective replacement of 

20 objectionable content: (1) that can allow the user or viewer to watch a program, while 
replacing objectionable material (audio or video) with less-objectionable material, such 
as, §lcl advertisements; (2) that can mark specific portions of the audio and/or video 
content of a program as objectionable for selective and precise replacement of the 
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audio and/or video signal; (3) that can identify specific portions of the audio and/or video 
content of a program as objectionable for selective and precise replacement of the 
audio and/or video signal; (4) that can ensure that only the objectionable content is 
replaced with less-objectionable material, such as, e^ advertisements, and that all 
non-objectionable content is not replaced and all objectionable content is replaced; (5) 
that can replace both audio and video content that is objectionable with less- 
objectionable material, such as, e^ advertisements; (6) that can mark, identify, and 
replace objectionable content based on different types and levels of intensity of 
objectionable material; and (7) that can be implemented in a variety of audio-video 
devices without requiring any specialized hardware or additional components. 

SUMMARY OF THE INVENTION 
[0016] The primary object of the present invention is to overcome the deficiencies of 
the prior art described above by providing a system, method, and computer program 
product that can selectively replace objectionable material (audio or video) with less- 
objectionable material, such as, e^ advertisements. 

[0017] Another key object of the present invention is to provide a system, method, 
and computer program product than can allow the user or viewer to watch a program, 
while replacing objectionable material (audio or video) with less-objectionable material, 
such as, e.g. advertisements. 

[0018] Still another key object of the present invention is to provide a system, method, 
and computer program product that can mark specific portions of the audio and/or video 
content of a program as objectionable for selective and precise replacement of the 
audio and/or video signal with less-objectionable material, such as, e.g. advertisements. 



[0019] Yet another key object of the present invention is to provide a system, method, 
and computer program product that can identify specific portions of the audio and/or 
video content of a program as objectionable for selective and precise replacement of 
the audio and/or video signal with less-objectionable material, such as, ex^ 
5 advertisements. 

[0020] Another key object of the present invention is to provide a system, method, 
and computer program product that can ensure that only the objectionable content is 
replaced, and that substantially all non-objectionable content is not replaced and 
substantially all objectionable content is replaced with less-objectionable material, such 

10 as, a<L advertisements. 

[0021] Still another key object of the present invention is to provide a system, method, 
and computer program product that can replace both audio and video content that is 
objectionable with less-objectionable material, such as, ax^ advertisements. 
[0022] Yet another key object of the present invention is to provide a system, method, 

15 and computer program product that can mark, identify, and replace objectionable 
content based on different types and levels of intensity of objectionable material. 
[0023] Another key object of the present invention is to provide a system, method, 
and computer program product that can be implemented in a variety of audio-video 
devices without requiring any specialized hardware or additional components. 

20 [0024] The present invention achieves these objects and others by providing a 
system, method, and computer program product for the selective replacement of 
objectionable material (audio or video) with less-objectionable material, such as, e^ 
advertisements. The selective replacement objectionable content in a program is 
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accomplished by applying an encoding process and a decoding process to the audio 
and/or video signal of the program. The encoding process includes marking potentially 
objectionable material in the program with replacement information that identifies the 
type (e.g., audio, violent content, sexual content, etc.) and intensity level (e.g., mild, 
5 graphic, extreme, etc.) of the potentially objectionable material using replacement 
codes. The decoding process includes comparing, during the presentation of the 
program to the user and prior to outputting the audio or displaying the video, the 
replacement information to a replacement criterion, which may include user supplied 
replacement settings, to determine whether replacement of the potentially objectionable 

10 content (i.e., audio and/or video) with less-objectionable material, such as, e.g. 
advertisements should be performed. If replacement is to be performed, the 
objectionable content in the program is replaced according to the replacement 
information which, in the preferred embodiment, identifies the start and duration of the 
replacement and, for replacement of the video of the program, also identifies the display 

1 5 area of the video to be replaced. 

[0025] In a preferred embodiment of the present invention, the program is encoded 
with replacement information in the form of replacement codes that mark content that is 
potentially objectionable. In the case of an audio and/or video signal that is a television 
broadcast signal, for example, the replacement codes are encoded into the vertical 

20 blanking interval ("VBI") of the audio and/or video signal. 

[0026] The encoding process of the present invention is performed using an encoding 
system that includes a computer system and an encoding device. The encoded 
program is decoded using a replacement device, which receives and decodes the 
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program. The replacement device includes an extraction device that extracts the 
replacement codes from the audio and/or video signal. The extracted replacement 
codes are supplied to a processor that, based on the replacement codes and the user 
supplied replacement criterion, determines whether the audio and/or video should be 
5 replaced. 

[0027] If replacement of the audio is to be performed, the processor causes an audio 
replacement device to replace the portion of the audio containing the objectionable 
content with less-objectionable material, such as, e^ advertisements. The closed 
caption corresponding to the portion of the audio containing the objectionable content 

10 may also be replaced. If replacement of the video is to be performed, the processor 

causes a video replacement device to replace a specified area of the video display at a 
specified frame for a specified duration to thereby replace the objectionable portion of 
the video of the program with less-objectionable material, such as, ag. advertisements. 
[0028] Further features and advantages of the present invention, as well as the 

15 structure and operation of various embodiments of the present invention, are described 
in detail below with reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0029] The accompanying drawings, which are incorporated herein and form part of 
the specification, illustrate various embodiments of the present invention and, together 

20 with the description, further serve to explain the principles of the invention and to enable 
a person skilled in the pertinent art to make and use the invention. In the drawings, like 
reference numbers indicate identical or functionally similar elements. 
[0030] A more complete appreciation of the invention and many of the attendant 
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advantages thereof will be readily obtained as the same becomes better understood by 
reference to the following detailed description when considered in connection with the 
accompanying drawings, wherein: 

FIG. 1 is a flow diagram representing the method steps for selectively replacing 
objectionable content in a program with less-objectionable material, such as, ac^ 
advertisements in accordance with the system, method, and computer program product 
of the present invention; 

FIG. 2 is a flow diagram representing the method steps of the encoding process 
in accordance with the system, method, and computer program product of the present 
invention for selectively replacing objectionable content in a program with less- 
objectionable material, such as, ac^ advertisements; 

FIG. 3 is a flow diagram representing the decoding process in accordance with 
the system, method, and computer program product of the present invention for 
selectively replacing objectionable content in a program with less-objectionable material, 
such as, e^L advertisements; 

FIG. 4 is a functional block diagram of the encoding system of the system, 
method, and computer program product of the present invention for selectively replacing 
objectionable content in a program with less-objectionable material, such as, ac^ 
advertisements; and 

FIG. 5 is a functional block diagram of the replacement device of the system, 
method, and computer program product of the present invention for selectively replacing 
objectionable content in a program with less-objectionable material, such as, e.g. 
advertisements. 

10 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0031] In the following description, for purposes of explanation and not limitation, 
specific details are set forth, such as particular audio-video devices, audio and/or video 
signals, programs, processors, software, devices, components, circuits, techniques, 
data protocols, captioning formats, software products and systems, interfaces, 
hardware, etc. in order to provide a thorough understanding of the present invention. . 
However, it will be apparent to one skilled in the art that the present invention may be 
practiced in other embodiments that depart from these specific details. Detailed 
descriptions of well-known audio-video devices, audio and/or video signals, programs, 
processors, software, devices, components, circuits, techniques, data protocols, 
captioning formats, software products and systems, interfaces, hardware, etc. are 
omitted so as not to obscure the description of the present invention. 
[0032] As used herein, the term "audio-video device" may be intended to refer to any 
device adapted to receive an audio and/or video signal, including but not limited to, a 
set-top box (STB), television, video cassette recorder (VCR), digital video recorder 
(DVR), radio receiver, personal computer, digital video device (DVD), holographic 
display, or like devices and components. 

[0033] As used herein, the term "audio and/or video signal" may be intended to refer 
to a composite audio-video signal, a coded portion of a composite audio-video signal 
(e.g., closed caption component), a separate audio signal, or a separate video signal. 
Such signals may be analog signals, digital signals, streaming audio or video signals, 
broadcast signals, broadband signals, optical signals, circuit- or packet-switched 
signals, multicast or unicast signals. The audio and/or video signal referred to herein 
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may be delivered to the audio-video device by satellite, by cable, by VHF or UHF 
broadcast, by radio frequency (RF) transmission, telephone line, optical fiber, or any 
other wired or wireless means for transmitting an audio and/or video signal. The 
incoming audio and/or video signal may also be the output from an audio and/or video 
5 playback device, such as for example, a DVD player, video cassette recorder (VCR), 
digital video recorder (DVR), laser disc player, compact disc (CD) player, a holographic 
memory, or like devices and components. 

[0034] As used herein, the term "program" and "programming" may be intended to 
refer to an audio, audio-video, or video presentation such as, for example, a movie, a 
10 television program (e.g., a television show, movie, documentary, sporting event, news 
broadcast, etc.), radio broadcast, or any other broadcast (live or recorded) or recorded 
production. 

[0035] Replacement, when used in the context of replacement of the video, means 
replacing a portion of the video by, for example, replacing the video being displayed on 

15 the display screen with less-objectionable material, such as, e^ advertisements. The 
advertisement may be, e^ a depiction of a product or service such as, e^ 'an ACME 
anti-gravity device', or a commercial advertising a product or service. 
[0036] Replacement, when used in the context of replacement of the audio, means 
replacing a portion of the audio by, for example, replacing the audio being displayed on 

20 the display screen with less-objectionable material, such as, ex^ advertisements. The 
advertisement may be, ac^ the name of a product or service such as, ex^ 'ACME 
skydiving', or a commercial advertising a product or service. 

[0037] Additional terminology, conventional technologies associated with audio-video 
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devices, audio and/or video signals, programming, and closed captioning, as well as 
various standards, conventions, and equipment are set forth in "Inside Captioning" by 
Gary D. Robson, Cyber Dawg Publishing (1997), the contents of which are hereby 
incorporated herein by reference. 
I. Method of the Present Invention 

[0038] The system, method, and computer program product of the present invention 
provides for the selective replacement of objectionable content in a program with less- 
objectionable material, such as, ag^ advertisements. Programming, and the audio 
and/or video signals representing such programming, that the system, method, and 
computer program product of the present invention may be employed to replace 
includes NTSC, PAL, and SECAM video, streaming audio/video, and any other analog 
or digitally represented programming. 

[0039] With reference to FIG. 1 , a flow diagram representing the method steps for 
selectively replacing objectionable content in a program with less-objectionable material, 
such as, e& advertisements in accordance with the system, method, and computer 
program product of the present invention is shown. The method of the present 
invention may comprise the step 1 10 of encoding the program to mark potentially 
objectionable content present in the program. The encoding step 110 may include 
embedding or inserting replacement information into the program sufficient to identify 
the potentially objectionable content (video and/or audio) or may include storing or 
transmitting replacement information, separately from the program, sufficient to identify 
the potentially objectionable content in the program. As discussed in greater detail 
below, step 110 may preferably be performed off-line, which involves performing the 
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encoding after production of the program, but prior to its broadcast, playback, etc. 
Alternatively, the encoding step 110 may be performed in real-time or on-line as the 
program is being broadcast, played back, etc. The encoding step 110 may be 
performed by a broadcaster, captioner, production house, post production house, or 
5 other entity. 

[0040] At step 120, the encoded program, which may be received by the audio-video 
device, may be decoded. The encoded program may be received by the audio-video 
device in the form of an audio and/or video signal representing the encoded program. 
The decoding process extracts the encoded replacement information from the audio 
10 and/or video signal. As described above, the replacement information may be in the 
VBI or, alternatively, may be stored in a database or file transmitted separately to the 
audio-video device, or embedded in another part of the incoming audio and/or video 
signal for digital signals. 

[0041] The encoded program is, preferably, received by the replacement device, 
15 which decodes the encoded program. The encoded program, however, may be 

provided to the replacement device through any suitable means such as being supplied 
on a recorded medium, such as a video cassette, DVD, or CD for playback by a suitable 
player for decoding. 

[0042] The extracted replacement information, which includes information identifying 
20 potentially objectionable material in the program, may be then compared with the 

replacement criterion at step 130. The replacement criterion may include information 
that may be provided by the user, supplied by the manufacturer of the replacement 
device, and/or periodically supplied by a remote source (e.g., a cable company) and 
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indicates the type and level of potentially objectionable material that is to be replaced in 
the video as well as the particular words and phrases that are to be replaced in the 
audio. 

[0043] If the replacement information does not match the replacement criterion then 
no replacement is performed. If the replacement information matches the replacement 
criterion, then the program material may be replaced according to the replacement 
information at step 140. Thus, the replacement information also includes information 
sufficient to permit the replacement device to replace the objectionable content with 
less-objectionable content, such as information sufficient to locate the content (either 
temporally, spatially, or positionally). 

[0044] The less-objectionable content may be simply substituted for the objectionable 
content. The less-objectionable content may be received by the audio-video device in 
the form of an audio and/or video signal representing the less-objectionable content. 
The less-objectionable content may, however, be provided to the replacement device 
through any suitable means such as being supplied on a recorded medium, such as a 
video cassette, DVD, or CD for playback by a suitable player. A switch, ag^ may be 
used to switch the input from the stream of objectionable content to that of less- 
objectionable content. 

[0045] Finally, the replaced program may then be produced as an output for the 
immediate presentation to the viewer or for recording or transmission at step 1 50. 

A. Replacement Information of the Present Invention 
[0046] In one embodiment of the present invention, the replacement information 
includes any one of three replacement codes, which indicate the type of potentially 
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objectionable material that may be present, as well as information sufficient to 
selectively and accurately perform replacement of the material. 

1. A-Codes (Audio) 

[0047] Replacement code "A" (A-code or audio code) represents "Audio" and marks 
(e.g., identifies the placement in the program) a potentially objectionable word or phrase 
present in the audio portion of the program. As described in more detail below, each A- , 
code consists of a preamble containing the word being identified as potentially 
objectionable, which permits users to program their replacement device to replace (or 
not replace) on a word-by-word basis. The A-code also includes a duration, and a start 
symbol. 

2. V-Codes (Violent content) 

[0048] Replacement code "V" (V-code or violence code) represents "Violence" and 
marks potentially objectionable violent material present in the video portion of the 
program. Each V-code contains information relating to the level of intensity of the 
violence referred to in this example embodiment as an intensity level. The intensity 
levels in this example embodiment are: 

1 - Mild or fantasy violence 

2 - Graphic violence 

3 - Extreme violence or rape 

[0049] In addition, each V-code includes information relating to a set of display 
coordinates to be replaced, a start code, and duration information. 

3. S-Codes (Sexual Content) 

[0050] Replacement code "S" (S-code or sex code) marks potentially objectionable 
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sexual content in the video portion of the program. S-codes and V-codes function 
nearly identically and are differentiated primarily to allow end users to replace S-codes 
independently of V-codes, and vice versa. S-codes have intensity levels as follows: 

1 - Mild nudity (topless, see-through) 

2 - Full nudity or highly suggestive sexual behavior 

3 - Graphic sex or strong sexual content 

[0051] The content that may be considered "potentially objectionable" may be defined 
by the designer and will vary according to the implementation of the present invention. 
Alternate embodiments may use additional and or different codes to permit other types 
of replacement. For example, a N-code could be used to identify and replace nudity, 
but would not identify or replace other sexually explicit content (e.g., kissing). Likewise, 
an R-code might be used to identify and replace potentially objectionable religious 
content. Other embodiments might use altogether different replacement codes, 
replacement information and/or syntax. 

4. Replacement Code Syntax 
[0052] All of the replacement codes share a common syntax, although some of the 
attributes are not present in all code types. The following replacement code syntax may 
be used in the encoding and decoding process of the preferred embodiment. The 
general format is presented below with lowercase words and letters being included as 
shown, and UPPERCASE words representing the attributes to be inserted by the 
encoding system. 

[0053] The format for the codes is as follows: 
replacement: 7YPE>[w:l/l/0^ 
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HECKSUM][s] 

[0054] As shown above, the start of the replacement code may be enclosed in angle 
brackets, and uses the keyword "replacement" to distinguish it from a URL (as 
described in EIA-608-B §7.10 through 7.14, which will typically begin with http). The 
"TYPE" portion of the code (following the word "replacement:") will be the letter a, s, or v 
in this example embodiment. If the two characters after "<replacement:" are anything 
other than "a>", "s>", or "v>", then the replacement code may be discarded. 
[0055] Following the end bracket ">" is a list of attributes associated with the 
replacement type. All of the attributes shown above except channel and word are 
required for S-codes and V-codes. All except intensity and area are required for A- 
codes. 

a. WORD Attribute 
[0056] The WORD attribute may be used for A-codes only and identifies the actual 
word being marked for potential replacement, rendered in all uppercase. If the word 
being inserted in the WORD attribute contains accent marks, they are stripped out in the 
capitalization process prior to insertion, which renders the use of extended characters 
unnecessary. 

[0057] Since many televisions and decoders (i.e., set top cable boxes) allow viewers 
to see Text2 data, it may be preferable to prevent the word being replaced from 
appearing in plain text. Therefore, in the preferred embodiment the word or phrase may 
be encoded prior to being inserting as an attribute. 

[0058] In this embodiment the word may be encoded using a simple ROT-13 code as 
may be well-known in the art. In ROT-13 encoding, each alphabetic character may be 
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replaced by the character that follows 13 positions later - wrapping around if necessary. 
"A" becomes "N", "B" becomes "0", and so forth. Non-alphabetic characters are not 
modified. The word being encoded may not include a right square bracket character 
("]"). but may contain other non-alphabetic characters such as numbers, apostrophes, 
5 and other punctuation marks. 

[0059] ROT-13 encoding may be used because it may be self-reversing (applying 
ROT-13 again brings back the original plaintext), easy to encode, fast executing, and 
adequately obscures the text. However, any suitable encoding method would suffice. 
The following Visual Basic routine provides an example of a ROT-13 encoding software 
10 routine: 



Private Function Rot13(lnText As String) As String 



Dim i As Integer 



Dim OutText As String 
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Dim ThisChar As Integer 



InText = UCase(lnText) 



Change to all uppercase 



OutText = 



mi 



For i = 1 To Len(lnText) 



' Loop through input text 



ThisChar = Asc(Mid(lnText, i, 1)) 
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If (ThisChar >= AscfA")) And (ThisChar <= Asc("Z")) Then 



It's an alphabetic character - rotate it 



If ThisChar < Asc("N") Then 



OutText = OutText & ChnThisChar + 13) 
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Else 

OutText = OutText & Chr(ThisChar - 13) 
End If 
Else 

' It's non-alphabetic - include as-is 
OutText = OutText & Chr(ThisChar) 
End If 
Next i 

Rot1 3 = OutText 
End Function 



b. CHANNEL Attribute 

This CHANNEL attribute may be used only for A-codes. NTSC television as used in 
North America allows for a primary audio program and a second audio program (SAP), 
which can be another language or a descriptive audio channel for blind viewers. To 
indicate that the word may be present in the primary audio, the channel attribute may be 
represented as a "P" (for example [c:P]) or the attribute may be omitted entirely with the 
default being the primary audio. Conversely, to indicate that the word may be present in 
the secondary audio program the attribute may be set to "S" ([c:S]). 

c. INTENSITY Attribute 

[0060] The INTENSITY attribute may be used with V-codes and S-codes only and 
represents the level of intensity of the violent or sexual content in the program, as 
described above. The attribute value may be specified as a single digit (e.g., [i:2] 
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representing level 2 violent content or sexual content). 
[0061] The valid range of intensities may be 1-3. Level 0 is not used and the 
replacement device interprets it as equivalent to omitting the replacement code entirely. 
[0062] If the replacement device receives a replacement code with an intensity 
5 attribute greater than 3, it treats the replacement code as a level 3 intensity. If the 

replacement device receives an INTENSITY attribute that is not a valid positive integer, 
it ignores the replacement code entirely and performs no replacement. For example: 
[i:0] Ignored; 
[i:329] Treated as level 3; 

10 [i:-1] Ignored; 
[i:two] Ignored. 

d. DURATION Attribute 
[0063] The DURATION attribute, which may be required for all codes, specifies the 
number of frames (or alternately, thirtieths of a second) of the program which are to be 

1 5 replaced from the reception of the START attribute (discussed below). In the case of A- 
codes, it specifies how long the audio will be replaced. For V-codes and S-codes, the 
DURATION attribute specifies how long a portion of the screen will be replaced. 
The valid range of the DURATION attribute may be 1 to 300 frames (1/30 second to ten 
seconds). If replacement of more than ten seconds is required, a second replacement 

20 code must be used. This puts a maximum limit on the amount of programming that will 
be replaced by an erroneous or poorly-coded replacement code. 
[0064] If a replacement device receives a replacement code with a DURATION 
attribute greater than three hundred frames, it treats the replacement code as having a 
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three hundred frame DURATION. If it receives a DURATION attribute that is not a valid 
positive integer, it ignores the replacement code entirely and does not perform any 
replacement. 

[0065] In this embodiment, the DURATION attribute identifies the number of frames 
5 from the frame in which the start signal may be found. However, in alternate 

embodiments the DURATION attribute could identify an absolute stop time, a relative 
stop time, a duration time, or a stop location within the signal or recording. 

e. AREA Attribute 
[0066] The AREA attribute may be used in V-codes and S-codes only and specifies 

10 the television display coordinates of the area to be replaced. The format used may be 
[a:x1 ,y1-x2,y2]. The upper left corner of the screen may be coordinate 0,0 and the 
lower right may be 99,99. The AREA attribute may always be expressed so that x1 may 
be less than x2 and y1 may be less than y2. As an example, to block coordinates 
(10,20) to (40,60), the attribute would read [a:10,20-40,60]. 

1 5 [0067] The coordinates are relative to the full video picture in this example 

embodiment and not just the viewable area or safe title area. Because of the way 
televisions are designed and manufactured, the edges of the picture are not visible on 
all televisions. Consequently, all calculations of the AREA attribute are measured from 
the actual edges of the transmitted video. 

20 [0068] Different implementations of replacement devices of the present invention may 
use different methods for generating the replacement area. Some implementations may 
use the caption decoding chip to generate a background while others may use an on- 
screen display (OSD) chip. Still others may use custom circuitry. 
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[0069] This means that replacement devices may be constrained to different 
coordinate systems depending on how the replacement rectangle is generated. For 
example, caption decoder chips often may not be capable of on-screen graphics (for 
replacement) close to the edge of the screen. 
5 [0070] This also means that the video safe area and safe title area will vary between 
replacement devices. An OSD chip, for example, may be able to generate on-screen 
graphics considerably closer to the edge of the picture than a decoder chip. 
[0071] If the replacement device receives an AREA attribute that goes outside of the 
area that it can block, then it may be possible that objectionable material may be visible 

10 around the outside edge of the picture. If the replacement device has the ability to 
generate a complete screen, it does so instead of generating an obscuring rectangle 
whenever this situation arises. In other words, if the AREA attribute includes screen 
area outside the portion of the screen that the replacement device can block, the 
replacement device blocks the video signal in its entirety. This ensures that everything 

15 that is supposed to be replaced may be replaced. This may be accomplished by, e.g. 
tiling. 

[0072] The replacement device generates the smallest replacement rectangle that 
completely covers the area to be replaced. This means that when converting from the 
replacement code coordinate system to the replacement device coordinate system, x1 
20 and y1 should always be rounded down, and x2 and y2 should always be rounded up. 
A rectangle that displays less-objectionable material over the replaced area may be 
preferred to blurring or pixelating as the displayed rectangle guarantees total 
obfuscation of the specified area. However, alternate embodiments of the present 
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invention may use other methods of obfuscating the replaced area (which may include 
replacement the entire screen). 

[0073] If any of the four numbers of the AREA attribute (x1 , y1 , x2, or y2) are outside 
of the range 0-99, or if x1 is greater than or equal to x2, or if y1 is greater than or equal 
5 to y2, or if the attribute is syntactically invalid, then the partially-completed replacement 
code may be ignored, and all incoming characters are ignored until the next bracket 
character ("<") is received. 

f. START Attribute 
[0074] All replacement codes require the START attribute. Note that the START 
1 0 attribute follows the checksum at the end of the replacement code. This allows precise 
frame accurate placement of the START attribute after the replacement device may 
have had time to process the entire replacement code. 

[0075] Replacement will begin in the frame containing the closing bracket of the 
START attribute and will continue for the number of frames specified by the DURATION 

15 attribute. However, in alternate embodiments the START attribute could identify an 

absolute start time, a relative start time, or a start location within the signal or recording. 
When encoding Line 21 data, CC1 and CC2 are given precedence over Textl and 
Text2. Thus, it may be important that the encoding system place the start attribute at 
the appropriate time, even if the rest of the replacement code may have to be shifted 

20 several seconds earlier. The replacement device of the preferred embodiment will 

tolerate a delay of up to ten seconds between the checksum and the START attribute, 
which allows the broadcasting system maximum flexibility in fitting the data from CC1, 
CC2, and Textl around the code in Text2. If more than ten seconds elapses between 
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the closing bracket (">") on a replacement code and the start attribute, then the 
currently received replacement code and further START attributes are ignored until 
another replacement code may be received. 

[0076] When there may be no partially-completed replacement code pending, all 
5 characters in Text2 are ignored by the replacement device until a left angle bracket ("<") 
may be detected. If the seven characters immediately following the left angle bracket 
are anything other than "replacement:", then the data may be discarded (ignored) and 
the replacement device returns to ignoring anything other than "<" characters. If a 
replacement code is missing a required attribute (such as the duration), the code may 
10 be ignored. 

g. Checksums 

[0077] The checksum may be computed using all data in the code from the opening 
angle bracket character ("<") to the closing square bracket character ("]")of the last 
attribute not including the checksum itself or the start code. The calculation may be 
15 performed according to the algorithm Internet RFC 1071 , which makes it compatible 
with URL transmissions as defined in EIA-608-B. 

[0078] The checksum may be computed by combining adjacent characters in the 
code into 16-bit integers and taking a ones-complement sum. If there is an odd number 
of characters, the final character may be paired with a zero. The ones-complement of 
20 this sum may be expressed as a four-digit hexadecimal number in square brackets. 

[0079] The following Visual Basic routine provides an example of a software routine 
for calculating the checksum: 
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Private Function CalculateChecksum(code as String) as String 
Dim i As Integer 
Dim CS As Long 
i= 1 

5 CS = 0 

' Build the checksum using long-integer arithmetic 
Do While i <= Len(code) 
If i + 1 <= Len(code) Then 
CS = CS + Asc(Mid(code, i + 1 , 1 )) 
10 End If 

CS = CS + (256 * Asc(Mid(code, i, 1))) 
i = i + 2 
Loop 

1 Convert to ones-complement 

15 CS = (CS Mod 65536) + ((CS And &HFFFF0000) / 65536) 

r Invert 

CS = 65535 - CS 
1 Return checksum as string 
CalculateChecksum = "[" & Hex(CS) & "]" 
20 End Function 



[0080] If the computed checksum for a code does not match the received checksum, 
then the next start attribute may be ignored, and the partially-completed code (or 
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completed code) may be discarded. This prevents aberrant behavior when poor 
television reception causes transmission errors in the replacement codes. 

h. Example Replacement Codes 
[0081] The following are examples and interpretations of transmitted replacement 
5 codes and their attributes. 

Example 1 

<replacement:v>[i:2][d:30][a:50,0-99,99][5658][s] 
<replacement:v> - This is a V-code (violence); 
10 [i:2] - The intensity is 2 (graphic violence); 

[d:30] - The duration is 30 frames (one full second); 

[a:50,0-99,99] - The area of the screen to block is the full right half of the screen, from 
coordinates (50,0) at the top center to (99,99) at the bottom right; 
[5658] - This is the checksum; 
15 [s]- Start. 

Example 2 

<replacement:a>[w:QNZA][c:P][d:10][0755][s] 
<replacement:a> - This is an A-code (audio); 
20 [w:QNZA] - The word being marked is "damn". Shifting to all uppercase and applying 
ROT-1 3 coding encodes it into QNZA; 
[c:P] - The audio being replaced is the primary program audio; 
[d:10] - The duration is 10 frames (1/3 second); 
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[0755][s] - Checksum and start. 
Example 3 

<repiacement:s>[i:1][d:75][a:25,25-75,75][86F5][s] 
5 <replacement:s> - This is an S-code (sexual content); 
[i:1] - The intensity is 1 (mild nudity); 
[d:75] - The duration is 75 frames (2-1/2 seconds); 

[a:25,25-75,75] - The area being obscured is half the screen width and half the screen 
height, right in the center of the screen; 
1 0 [86F5][s] - Checksum and start. 

Example 4 

<replacement:a>[w:ZVREQN][c:S][d:15][ACFA][s] 
<replacement:a> - This is an A-code indicating the audio is to be replaced; 
15 [w.ZVREQN] - The word being marked is mierda (a Spanish obscenity). 

[c:S] - The audio channel to potentially be replaced is the SAP (second audio program) 
channel; 

[d:15] - The duration is 15 frames (1/2 second); 
[ACFA][s] - Checksum and start. 
20 [0082] In the present embodiment, characters used for replacement codes in Text2 
are generated and interpreted as ISO-8859-1 characters (also known as Latin-1 and 
compatible with US-ASCII), rather than as E1A/CEA-608-B characters. This maintains 
compatibility with URLs transmitted in Text2. Only characters between 20h and 7Eh are 
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used. Two-byte EIA-608-B character codes and control codes are not permitted. 

B. Encoding Process 
[0083] As discussed above, the present invention includes the step of encoding the 
program to mark potentially objectionable material in the program. The encoding 
5 process generates replacement information and marks potentially objectionable content 
of the program, which may be subsequently used to identify and replace objectionable 
content from the program. The encoding may be performed off-line or on-line by the 
broadcaster, captioner, production house, post production house, or any entity with 
access to data representing the program, such as access to an audio and/or video 

10 signal, transmission signal, or recording of the program. The encoding may include 
embedding or inserting replacement information into the program sufficient to identify 
the potentially objectionable material (video and/or audio) or may include storing 
replacement information in a database or transmitting replacement information in a file 
separately from the program, sufficient to identify the potentially objectionable material 

15 in the program. 

[0084] The encoding process may be accomplished through the use of an encoding 
system, which includes a computer system appropriately programmed to implement the 
encoding process of the present invention, as described in greater detail below. 
As may be well-known in the art, a conventional television signal includes five hundred 

20 and twenty-five (525) lines of information. However, not all of these lines of information 
are presented visually to the viewer. Specifically, line twenty-one may include the 
closed caption signal if it is supplied by the broadcaster. 

[0085] In one embodiment of the present invention, the replacement information may 

29 



be encoded in the Text 2 area of Line 21 of the VBI of a conventional television signal. 
In North America, the VBI may comprise lines 10 through 21 of the video signal. 
Transmission of data in line 21 of the video signal may be defined by Electronic 
Industries Association specification No. EIA-608, which may be incorporated herein by 
5 reference. More specifically, specification EIA-608 specifies protocols for embedding 
digitally encoded information packets containing information about a television program. 
EIA-608 thus defines protocols for transmitting information that may be useful for 
practicing the present invention. 

[0086] In a digitally represented program, the data may be included in data packets 
10 having a particular header or other identifying characteristic. Alternately, the 

replacement information may be included as a separate data stream or data file and 
that, among other information, contains information identifying the frames or packets of 
data that contain potentially objectionable material in the program. 
[0087] Referring now to FIG. 2, a flow diagram representing the method steps of the 
15 encoding process in accordance with the system, method, and computer program 

product of the present invention for selectively replacement objectionable content from a 
program is shown. The encoding process 200 encodes the replacement information 
that identifies potentially objectionable video content according to the method steps 
shown in FIG. 2. At step 210, an encoder (e.g., a person) watches and listens to the 
20 program to identify potentially objectionable material in the program. As discussed, in 
one embodiment of the present invention, sexual content and violent content are 
considered potentially objectionable video material, and particular words and phrases, 
such as, for example, foul language like curse words, are also considered potentially 
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objectionable audio material. 

[0088] Once the potentially objectionable content is identified at step 210, the start 
and stop of the potentially objectionable material may be identified at step 220. This 
information might be the start frame and stop frame of the material, the start frame and 
5 duration of the material, start time and stop time (as measured from the beginning of the 
program), or any information suitable to facilitate reasonably precise replacement of the 
program, in the preferred embodiment of the present invention, the encoder identifies 
the frames in which the potentially objectionable material may be present, which may be 
stored in the memory of the encoding system and used to determine the DURATION 

10 and START attributes of the replacement code, as described above. 

[0089] At step 230, the system determines whether the potentially objectionable 
content identified may be video content. If it is, then at step 240, the area of the video 
display where the potentially objectionable video content may be present and potentially 
to be replaced may be identified. The area of the display containing the potentially 

15 objectionable material may be identified by the encoder by using the pointing device 
(e.g., the mouse) of the encoding system to draw a rectangle (or other arbitrary shape) 
around the outer boundary of the potentially objectionable content. The coordinates of 
the shape are then stored in the encoding system memory to generate the AREA 
attribute of the replacement code. 

20 [0090] If the potentially objectionable content identified may be video content, the 

intensity level of the potentially objectionable video material may be defined at step 250. 
As discussed above, in the example embodiment, the intensity level may be 1, 2, or 3 
and may be dependent on the type of potentially objectionable video material (e.g., 
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sexual content or violent content) and the intensity of the identified portion of the 
program. Thus, after comparing the potentially objectionable material with the 
appropriate intensity level standard, the encoder inputs the appropriate intensity level 
into the encoding system, which may be stored in memory and used to generate the 
replacement code. 

[0091] If the potentially objectionable content identified may be audio content, the 
potentially objectionable audio material may be identified at step 260. In the preferred 
embodiment, this may be accomplished by supplying the word or phrase to the 
encoding computer system, which encodes the word according to ROT-13 encoding, as 
described above, and stores the encoded word in memory for use in the replacement 
code. If the potentially objectionable content identified may be audio content, the audio 
channel of the potentially objectionable audio may be identified at step 270 and stored 
in the encoding system memory for inclusion into the replacement code. 
[0092] At step 280, the type of potentially objectionable content may be defined. As 
discussed, in the preferred embodiment the potentially objectionable content may be 
identified as either sexual video content (S-code), violent video content (V-code), or 
audio content (A-code). The encoder supplies the type of potentially objectionable 
material to the encoding system, where it may be stored for use in the replacement 
code. 

[0093] Finally, at step 290, the replacement information may be applied to the 
program by marking the program with the replacement information. As discussed 
above, this may include embedding the information in the program or storing the 
replacement information to be supplied with the program in a separate signal, 
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transmission, or data file. In the preferred embodiment of the present invention, the 
replacement codes are encoded in the Text 2 area of Line 21 of the VBI. 
[0094] While the encoder of the example embodiment described above may be a 
person watching and listening to the program and supplying replacement information, in 
5 an alternate embodiment, the encoder may be a closed caption decoder and an 

appropriately programmed computer system that compares the closed caption of the 
program with a database of potentially objectionable words and phrases and provides 
inputs to the encoding system as may be appropriate. In still another embodiment, well- 
known voice recognition software that converts the audio to text could be used with an 

10 appropriately programmed computer system that compares the converted audio of the 
program with a database of potentially objectionable words and phrases. 

C. Decoding Process 
[0095] Upon being supplied the encoded program in the form of an audio and/or video 
signal representing the encoded program - either by reception of a transmission or by 

15 playback from a recorded medium - the replacement device performs the decoding 

process. With reference to FIG. 3, a flow diagram representing the decoding process in 
accordance with the system, method, and computer program product of the present 
invention for selectively replacement objectionable content from a program may be 
shown. 

20 [0096] As will be discussed in more detail below, the replacement device includes a 
processor executing software designed to operate the replacement device to implement 
the decoding steps. Appropriate software can be readily prepared by programmers of 
ordinary skill based on the teachings of the present disclosure, as will be apparent to 
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those skilled in the software arts. 

[0097] In one embodiment of the present invention, at step 310 of the decoding 
process 300, the closed caption of the video signal may be decoded. At step 320, the 
replacement codes are identified, which themselves identify potentially objectionable 
content. As discussed in more detail above, if the process receives a partial 
replacement code, a replacement code with invalid syntax, a replacement code with an 
incorrect checksum, or a replacement code with some other defect, the process will 
discard the replacement code. The combination of steps 310 and 320 of the decoding 
process of this example embodiment shown in FIG. 3, corresponds generally to step 
120 of FIG. 1 (Extract Replacement Information). 

[0098] In this embodiment, the codes may be either video or audio codes. Thus, at 
step 330, the process determines whether the potentially objectionable content may be 
video. If so, the process continues to step 340, which determines if the potentially 
objectionable material may be of a type that may be being replaced. More specifically, 
the replacement code may be compared with the replacement criterion that may be 
supplied by the user to determine whether the user desires to replacement that 
particular type (e.g., sexual content or violent content) of potentially objectionable 
material. 

[0099] If the replacement code does not match the replacement criterion (indicating 
that that type of potentially objectionable material should not be replaced), then no 
replacement need be performed and the process terminates at step 440 until new 
replacement information may be identified. For example, if the user may have elected 
to not replacement any violent content and the replacement code may be a V-code, 
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then the process does not replacement the potentially objectionable video content 
marked by the V-code. 

[0100] If, at step 340, it may be determined that the replacement code of the 
replacement information does match the replacement criterion, (indicating that the 
5 potentially objectionable content may be of a type that should be replaced), at step 350, 
the intensity level of the replacement code may be compared with a corresponding 
replacement criterion to determine if the material should be replaced. Specifically, the 
process determines if the intensity level of the potentially objectionable content may be 
of a level that should be replaced by comparing the intensity level of the material with 

10 the intensity level that the user supplied replacement criterion indicates should be 
replaced. For example, if the user supplied replacement criterion indicates that only 
sexual content with an intensity level of two or greater should be replaced, and the 
replacement code may be an S-code with an intensity level of one, the process does not 
replace the video content marked by the S-code. If it is determined at step 350 that the 

15 video should not be replaced, the process terminates at step 440 until new replacement 
information may be identified. 

[0101] If, at step 350, it may be determined that the intensity level of the potentially 
objectionable content may be of a level that should be replaced, the area of the video 
display to be replaced may be identified from the replacement code's AREA attribute at 
20 step 360, and at step 370, the start and stop points for replacement the video are 
determined by using the DURATION and START attributes as described above. 
[0102] At step 380, the video may be replaced according to the processed 
replacement information and, at step 420, may be output for viewing, transmission, or 
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recording. 

[0103] If, at step 330, the potentially objectionable content may be determined to be 
audio, the process continues to step 390, which determines if the word or phrase 
identified as potentially objectionable may be a word or phrase (hereinafter collectively 
referred to as "word") that should be replaced. This determination may be 
accomplished by comparing the word with the replacement criterion. In this example 
embodiment, the audio replacement criterion includes a word list, described in more 
detail below, including information of numerous variations of words that the user may 
have indicated should be replaced. As discussed above, the word list may be include 
data from the user, downloaded from a remote source, and/or originally supplied with 
the replacement device. If an A-code identifies a word that may not be in the word list, 
the word marked by that A-code may not be replaced from the audio, and the process 
terminates at step 440 until new replacement information may be identified. If the word 
identified matches the replacement criterion (indicating that it may be a word or phrase 
that should be replaced), the start and stop points for replacement the audio are 
determined at step 400 by using the DURATION and START attributes as described 
above. 

[0104] The system may also optionally replace the closed caption, which prevents 
display of the offensive word in the displayed closed caption. If a word identified may 
be in the word list, each letter in the word present in the closed caption will be replaced 
in the closed caption with one letter of the name of a product or a service. A list of 
names of products or services of various lengths could, e^ be maintained in the less- 
objectionable material so that a word of the same length as that of the word to be 
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replaced could be selected. 

[0105] At step 410, the audio may be replaced and at step 420 may be output for 
recording or for immediate audible production. 

[0106] Functional blocks within the dashed box 430 correspond generally to functional 
5 block 130 of FIG. 1. Likewise, the combination of functional blocks 380 and 410 of FIG. 
3 correspond generally to functional block 140 of FIG. 1 , and functional block 420 of 
FiG. 3 corresponds generally to block 150 of FIG. 1. 

D. Set-up Process 
[0107] As discussed above, the process of the present invention compares 

10 replacement information with the replacement criterion to determine whether potentially 
objectionable material is to be replaced with less-objectionable material. A replacement 
criterion may be supplied by the user, downloaded from a remote device, or supplied 
with the replacement device. The set up process allows the user to supply and modify a 
replacement criterion stored in the replacement device. In the preferred embodiment, 

15 the user enters replacement settings to establish and change a replacement criterion 
through a user input device, such as, for example, a remote control. 
[0108] The replacement settings entered by the user include the types of video 
replacement to be performed (violence and/or sexual content), if any, and the intensity 
level for each type of video replacement (e.g., 1 , 2, or 3). In an alternate embodiment, 

20 the user may enter a zero as the intensity level to indicate that no replacement should 
be performed. In this alternate embodiment, the decoding process need not perform 
step 340 (determine if code may be of a type to be replaced), but instead could simply 
compare the intensity level of the code with the replacement criterion. 
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[0109] In addition, the user may enter replacement settings to establish or change the 
replacement criterion for the audio, which includes information indicating whether audio 
replacement is to be performed and which may include specific words to be replaced. 
In addition, the replacement device of the present invention includes potentially 
5 objectionable words pre-stored during the manufacturing process and/or periodically 
downloaded from a remote source - all of which may be modified by the user. All the 
user-supplied information may be stored in non-volatile memory in the replacement 
device. 

[0110] In addition, the user can use the remote control to override the replacement 
10 function of the replacement device by establishing a password or personal identification 
number ("PIN"), and then entering the appropriate password or PIN to override the 
replacement process. This permits those who know the appropriate password or PIN to 
watch or listen to an individual program that the replacement device may otherwise 
replace portions of, without reprogramming the replacement device. At the end of the 
15 program (or after a time period), the replacement device may be programmed to 

automatically resume its normal replacement. Replacement device 600 can also detect 
when a program may be over by monitoring data packets containing the title or 
identification number for the current program, as may be well known in the art. Thus, in 
one embodiment of the present invention, the replacement device returns to normal 
20 replacement when a new program may be present (e.g., after the program may be over 
or when a new channel or station with a different program may be tuned to). 
II. System of the Present Invention 
A. Hardware 
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1. Encoding 

[0111] Referring now to FIG. 4, a functional block diagram of the encoding system of 
the system, method, and computer program product of the present invention for 
selectively replacement objectionable content from a program may be shown. This 
examplary embodiment of an encoding system in accordance with the present invention 
includes an encoding device 560 and a computer system 505 comprising a processor 
510 programmed according to the teachings of the present invention, which can be 
readily prepared by programmers of ordinary skill based on the teachings of the present 
disclosure as will be apparent to those skilled in the art. In addition, the computer 
system 505 includes a display device 520, an input device 530 (such as a keyboard and 
mouse), and a memory 540, including a database 550. The database contains, among 
other data, a word list of all the potentially objectionable words and phrases, and may 
be accessed by the system as part of the identification of potentially objectionable audio 
content. 

[0112] The processor 510 may be coupled to an encoding device 560, which receives 
the program from a media player 570, or that may be transmitted from a remote 
location. The encoding device encodes the program according to the encoding process 
of the present invention, as described above, and outputs the encoded program for 
storage in a media recorder 580 or for transmission. In addition, the system may 
include a video source 590, such as a DVD player or CD-ROM drive, that may be used 
to encode programs in other formats and/or mediums. 

[01 13] As discussed above, the encoding system of the preferred embodiment 
includes an encoding device 560 for encoding the replacement information into the 
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closed caption field of the VBI. There are numerous commercially available encoding 
devices that could be used to perform this step of the preferred embodiment such as the 
EEG EN-470 Smart Encoder III by EEG Enterprises, Inc. of Farmingdale, NY; the 
Closed Caption Encoder for Personal Computers (CCE/PC ) by Soft Touch Inc. of 
Alexandria, VA; many of the caption encoders manufactured by Ultech Co., Ltd. of 
Korea, and the Norpak TES3 (which can encode Line 21 captioning injo NTSC, PAL, 

and RFCAM x/idRO^ hv Nlornak of Kanata Ontario Canada In an alternafp pmhnHimpnt 

of the present invention, such as for encoding into a digital video signal, the Evertz 8070 
Digital Encoder by Evertz Microsystems of Burlington, Ontario, Canada could be used. 

2. Decoding 

[0114] Turning now to FIG. 5, a functional block diagram of the replacement device of 
the system, method, and computer program product of the present invention for 
selectively replacing objectionable content in a program with less-objectionable content 
such as, e^L advertising may be shown. Replacement device 600 includes a general 
purpose processor 610 appropriately programmed with software implementing the 
decoding process of the present invention that can be readily prepared by programmers 
of ordinary skill based on the teachings of the present disclosure, as will be apparent to 
those skilled in the software art. The processor 610 may be a microprocessor, 
microcontroller, RISC processor, or any processing device capable of executing 
software instructions and acting on them accordingly. 

[0115] The replacement device 600, in this example embodiment of the present 
invention, may be in the form of a set top box and includes an extraction device 660, an 
audio replacement device 640, a video replacement device 650, and main memory 620. 
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The main memory 620 includes random access memory (RAM) 630 or other dynamic 
storage device (e.g., dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM 
(SDRAM), flash RAM)), coupled to a bus for storing information and instructions to be 
executed by processor, in addition, main memory 620 also includes nonvolatile 
memory 640 for storing a replacement criterion, passwords/PINs, and other information. 
[0116] Replacement device 600 further includes a read only memory (ROM) or other 
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and electrically erasable PROM (EEPROM)) (not shown) coupled to a bus for storing 
static information and instructions for the processor 610. 

[01 17] The main memory 620 of the replacement device 600 may store structures, 
tables, records, or other data described herein. As discussed, the present invention 
includes software for controlling the replacement device, for driving a device or devices 
for implementing the decoding process of the present invention, and for enabling the 
replacement device to interact with a human user. Such software may include, but is 
not limited to, device drivers, operating systems, development tools, and application 
software. 

[0118] In the example embodiment of the present invention, a television broadcast 
signal may be received through a coaxial cable from the head end of the cable 
broadcaster. The incoming television signal could, alternatively, be delivered to the 
replacement device 600 by satellite, by cable, by VHF or UHF broadcast, or in any other 
suitable way. The incoming signal may also be the output from a video playback 
device, for example, a video cassette recorder ("VCR"), a laser disk player, or a DVD. 
Suitable transmission media includes coaxial cables, copper wire and fiber optics, 
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telephone transmission lines, and radio wave broadcasts including digital satellite 
broadcasts. The transmission media may also take the form of acoustic or light waves, 
such as those generated during radio wave and infrared data communications. 
[01 19] Similarly, less-objectionable material such as, e^ advertisements may be 
supplied in the form of a television broadcast signal received through a coaxial cable 
from the head end of the cable broadcaster. The less-objectionable material could, 
alternatively, be delivered to the replacement device 600 by satellite, by cable, by VHF 
or UHF broadcast, or in any other suitable way. The less-objectionable material may 
also be stored in a video playback device, for example, a video cassette recorder 
("VCR"), a laser disk player, or a DVD. Suitable transmission media includes coaxial 
cables, copper wire and fiber optics, telephone transmission lines, and radio wave 
broadcasts including digital satellite broadcasts. The transmission media may also take 
the form of acoustic or light waves, such as those generated during radio wave and 
infrared data communications. 

[0120] The replacement device 600 may have inputs for two or more signals, which 
comprise an incoming video FIRST VIDEO IN and/or audio FIRST AUDIO IN signal, 
and a signal representative of the less-objectionable material, SECOND VIDEO IN, 
SECOND AUDIO IN. The replacement device 600 also may have outputs for a 
composite video and audio, which may be connected to the appropriate inputs of a 
television, a VCR, or other device. Separate outputs and inputs (not shown) for a 
composite audio-video signal and/or other signal format(s) may also be provided in 
addition to, or instead of, the described outputs and inputs. 

[0121] Replacement device 600 also includes an extraction device 660. The video 
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signal may be supplied to the extraction device 660, which in the present examplary 
embodiment, may be a decoder chip, such as a 86129 Decoder and On Screen Display 
chip available from Zilog Corporation. The extraction device 660 could be implemented 
with other similar integrated circuits, with other multi-purpose integrated circuits, with 
discrete components, or with some combination thereof. The extraction device 660 
extracts the data from Line 21 of the VBI and supplies the data to the processor 610 for 
processing, thereby performing step 310 of the decoding process. 
[0122] The processor 610 receives the extracted data, groups the characters to 
recognized replacement codes, attributes, words, and delimiters whose meanings are 
discussed above, and performs the steps described above to determine whether to 
replace some of the program content with less-objectionable material. More 
specifically, after identifying a valid replacement code, the processor 610 compares the 
code and its associated replacement information (e.g., code type, and intensity level or 
word/phrase) with the stored replacement criterion to determine whether the potentially 
objectionable material should be replaced from the program, as discussed with respect 
to step 130 of FIG. 1 and the steps in block 430 of FIG. 3. 
[0123] Replacement device 600 also includes an audio replacement device 640, 
which may be operatively coupled to the processor 610. Upon determination that the 
audio may be to be replaced, the processor 610 causes the audio replacement device 
640 to replace the audio for a specified duration, and at the start time (or frame) as 
dictated by the attributes of the received A-code. The audio replacement device 640 
could be implemented with any of the many well-known devices capable of replacing the 
audio including discrete components (any of the numerous transistor switches), or 
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integrated circuit switches, which operate to disconnect the output from the standard 
input and reconnect it to an input of less-objectionable material. Other types of 
replacement devices well-known in the art would also suffice. 

[0124] Replacement device 600 also includes a video replacement device 650, which 
5 may be operatively coupled to the processor 610. When the received replacement code 
and the stored replacement criterion indicate that the video should be replaced, 
processor 610 causes the video replacement device 650 to replace the video in the 
area with less-objectionable material, at the start frame, and for the duration designated 
by the replacement code's attributes so that the objectionable video may not be 
10 produced by the television display or recorded on any medium receiving the video 
signal. In the preferred embodiment, the video replacement device 650 may be 
comprised of a switch which operates to disconnect the output from the standard input 
and reconnect it to an input of less-objectionable material to replace the video signal at 
the designated area. 

1 5 [0125] The video replacement device 650 could be implemented with any of the many 
well-known devices capable of replacing a portion of the video including character 
generators, superimposers, on-screen display chips, discrete components (e.g., 
transistor switches), other integrated circuit switches, or some combination thereof. 
Other types of video control devices well-known in the art would suffice as well. In 

20 addition, audio replacement 640 and video replacement device 650 could be 
implemented with the same or common circuitry. 

[0126] The output of the replacement device 600 may be supplied to a television, a 
monitor, or to a video cassette recorder (VCR) (or other audio/video display/recording 

44 



device) and then supplied to the television. 

[0127] Replacement device 600 also includes a user input device or component, 
which can be any means that allows a user to input information, such as replacement 
settings for establishing or changing a replacement criterion, including passwords, PINs, 
and other replacement information, to the replacement device. In the preferred 
embodiment of the present invention, the input means includes an infrared remote 
control transmitter (not shown), which transmits control signals to an infrared receiver 
(not shown) in the replacement device. When the infrared receiver receives a control 
signal from the infrared transmitter, it interrupts the processor 610. Processor 610 can 
then retrieve and interpret the transmitted information. The user input means allows the 
user to perform the set up process described above, as well as override replacement, 
and establish and change replacement settings. Replacement device 600 also includes 
a visual display, such as an LED display, to provide visual feedback to the user relating 
to its operation. 

[0128] The replacement device 600 performs a portion of, or may perform all of, the 
processing steps of the decoding process of the present invention in response to the 
processor 610 executing one or more sequences of one or more instructions contained 
in main memory 620. One or more processors in a multi-processing arrangement may 
also be employed to execute the sequences of instructions contained in the main 
memory. In alternative embodiments, hard-wired circuitry may be used in place of, or in 
combination with, software instructions. Thus, systems, devices, components, and 
methods embodying the present invention are not limited to any specific combination of 
hardware circuitry and software. 
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[0129] The replacement device 600 also includes a communication interface (not 
shown) coupled to the bus. The communication interface provides a two-way data 
communication coupling to a network link that may be connected to a local network 
(e.g., LAN), cable network, satellite system, etc. For example, the communication 
interface may be a network interface card to attach to any packet switched local area 
network (LAN). As another example, the communication interface may be an 
asymmetrical digital subscriber line (ADSL) card, an integrated services digital network 
(ISDN) card, or a modem to provide a data communication connection to a 
corresponding telephone line. Wireless links may also be implemented. In any such 
implementation, the communication interface sends and receives electrical, 
electromagnetic or optical signals that carry digital data streams representing various 
types of information. The replacement device can transmit notifications and receive 
data, including program code, through the network(s), network link, and communication 
interface. 

[0130] The replacement device may also include special purpose logic devices (e.g., 
application specific integrated circuits (ASICs)) or configurable logic devices (e.g., 
generic array of logic (GAL) or reprogrammable field programmable gate arrays 
(FPGAs)) as may be necessary for the intended application. 

[0131] The replacement device may optionally include a tuner, which, as may be well 
known in the art, receives a plurality of radio frequency signals. The tuner selects a 
particular channel (usually based on frequency), may or may not amplify it, and supplies 
it to the extracting device 610, video replacement 650 and/or audio replacement 640 of 
the replacement device 600 of the present invention. In an alternate embodiment, the 
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tuner may be a digital tuner that selects a digitally broadcast signal transmitted on a 
radio carrier wave. 

[0132] Those skilled in the art will readily understand that the functionality described 
herein can be achieved through the use of different components including the 
components used for the microprocessor and extraction device. The particular circuitry 
associated with the processor 610, and the circuitry for interfacing the processor to 
other devices, such as memory, and the other parts of the apparatus can have many 
variations. Those skilled in the art will readily understand how the replacement device 
600 could be constructed in light of this disclosure. 
B. Software 

[0133] The system and method of the present invention may be preferably 
implemented in general purpose processors running computer programs comprised of 
software modules implementing the process of the present invention for selectively 
replacing objectionable content in a program with less-objectionable content. The 
architecture, design, modules, and code of the software of the present invention could 
be implemented in a variety of ways and the manner in which it may be implemented 
may be largely a matter of design choice well within the ordinary skill level of those 
skilled in this art. Further, the data stored in memory and used by the computer 
program may be dependent on the software implementation of the present invention. 
As would be evident to one skilled in the art, the results of the computations for 
implementing the present invention, such as replacement code comparisons, 
replacement a criterion, passwords, PINs, replacement code attributes, and other 
information are stored in memory and retrieved from memory as needed. 
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1. Encoding Software 
[0134] As discussed, the present invention uses a computer system that interfaces 
with an encoding device to perform the encoding process according to the present 
invention. There are numerous commercially available computer programs that could 
be used to implement the present invention provided by companies such as Computer 
Prompting and Captioning Co. of Rockville, MD; Cheetah Systems, Inc. of Tucson, AZ; 
image Logic of Chevy Chase, MD; and Advantage Software of Stuart, FL, as well as 
proprietary software used by captioning service providers such as VITAC of 
Cannonsburg, PA; National Captioning Institute of Vienna, VA; and The Caption Center/ 
WGBH of Boston, MA. 

[0135J As will be apparent to those skilled in the software art, appropriately 
programmed software could also be readily prepared by programmers of ordinary skill 
based on the teachings of the present disclosure. 

2. Decoding Software 
[0136] The replacement device 600 performs processing steps of the present 
invention in response to the processor 610 executing one or more sequences of one or 
more instructions contained in main memory 620. As will be apparent to those skilled in 
the software art, appropriately programmed software can be readily prepared by 
programmers of ordinary skill based on the teachings of the present disclosure. 
[0137] In addition, a remote computer can load the instructions for implementing all or 
a portion of the present invention remotely into a dynamic memory and send the 
instructions over a telephone line using a modem or over a cable signal line to the bus 
in the replacement device 600. The bus carries the data to main memory, from which 
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the processor retrieves and executes the instructions. The instructions received by 
main memory may optionally be stored on storage device either before or after 
execution by processor. 

a. The Word List 
[0138] As discussed above, the replacement device 600 includes a replacement 
criterion, which are used to determine whether a marked portion of the program may be 
replaced. One portion of the replacement criterion may be the word list, which may be a 
set of data stored in the memory of the replacement device that identifies words and 
phrases that the user wishes to replace (provided audio replacement is turned on). The 
word list can be pre-programmed by the manufacturer, downloaded from a remote site, 
entered by the user, or some combination thereof which may include periodic updates. 
This word list may be stored in all uppercase with no accent marks. For example, an 
obscenity in another language with an accented lowercase a (such as a, a, a, or a) may 
be stored in the word list as an "A". The enya (N) may be stored as a plain N, and the c- 
cedilla (5) may be stored as a plain C. 

[0139] The words in the list may also contain wildcards. An asterisk (*) means zero or 
more of any character, and a question mark (?) means precisely one of any character. 
For example, if "*shit" is present in the word list, the processor will replace the words 
shit, bullshit, horseshit, and anything else ending with shit. If "shit*" is present, the 
device will replace the words shit, shitty, shithead, and anything else starting with "shit." 
If "*shit*" is present in the word list, the device will replace any word containing "shit" 
anywhere in it. 

[0140] The words before and after a hyphen are treated as separate words. For 
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example, if "shit" is in the word list, then the replacement device will replace and display 
"shit-for-brains" as, ac^ "ACME-for-brains." No wildcard is required to accomplish this. 
[0141] When words are being replaced in the closed caption, the entire word may be 
changed to characters - not just the non-wildcard parts. In the example above, if the list 
contains "*shit", then "bullshit" may be changed to, e^ "PEERLESS", not "bullACME." 
[0142] For replacement purposes of words in the closed caption, the apostrophe 
counts as the start of a new word. In other words, if "crap" is in the word list, the 
sentence That's a lotta crap" may be replaced to "That's a lotta ACME." 
[0143] While the examplary embodiment of the present invention described above 
uses V-codes, S-codes, and A-codes, additional codes or fewer codes be used. In 
addition, audio replacement may be omitted, or performed in a different manner. 
[0144] For example, Automatic Obscenity Audio Replacement (AOAR) may be a 
means for replacing the audio of programming that does not contain A-codes, which 
may be used instead of, or in addition to, A-codes. AOAR compares the text present in 
the closed caption that represents the audio with the word list to identify potentially 
objectionable audio. When AOAR is turned on and the replacement device recognizes 
a word that should replaced in the closed caption, it will estimate the time at which it 
should start replacing the audio signal, and replace the audio for a predetermined 
amount of time (usually 1.0 to 1.5 seconds) starting at the estimated start time, with 
less-objectionable content. 

[0145] One possible algorithm for estimating the time to replacement the audio 
assumes that a pop-on caption appears just as the caption's first word may be spoken. 
This assumption may not always be correct, but generally provides a good starting 
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point. The replacement device can either keep a running calculation of average reading 
rate over the prior few minutes, or simply use a typical value such as one hundred and 
sixty (160) words per minute. 

[0146] To determine when replacement should start, the system calculates how many 
5 words in the caption need to be skipped in order to reach the word to be replaced and 
multiplies the reading rate in frames per word by the number of words. 
[0147] For example, 160wpm equais 11.25 frames per word, if the objectionable 
word is the fourth word in the caption, the calculation would be 3 multiplied by 1 1 .25 
which equals 33.75 frames. Rounding this number to thirty-four (34), the processor 
1 0 would cause the audio replacement to start replacement thirty-four frames after the 
caption appears. 

[0148] As with the preferred embodiment, this alternate embodiment may replace 
each letter in the objectionable word with an uppercase letter of, ag^ an advertiser's 
name. This prevents display of the offensive word in the displayed closed caption. 

1 5 [0149] While the preferred embodiment may be implemented to replacement NTSC 
programming, an alternate embodiment could be used to replacement digital 
programming in which case the replacement information could be sent as a separate file 
(such as a separate database file) or with information in each digital packet (such as a 
portion of the header) that identifies the packet as replacement information. The 

20 replacement packets would then identify the program packets containing potentially 
objectionable content. 

[0150] In still another embodiment, the present invention could be used with High 
Definition TV (HDTV) in which the replacement information could be embedded in the 
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captions according to, exi EIA-708. 

[0151] In yet another embodiment, the replacement information could be stored in a 
separate file according to, e^ the Synchronized Accessible Media Interchange (SAMI) 
format. 

[0152] Information for practicing the invention could also be conveniently transmitted 
as, ej^ XDS data using the undefined data type provided by, e^ EIA-608. This or 
similar information could also be encoded and transmitted in ways which do not conform 
with the EIA-608 specification. 

[0153] The computer code of the present invention may be any interpreted or 
executable code mechanism, including but not limited to scripts, interpreters, dynamic 
link libraries, Java classes, and complete executable programs. In the example 
embodiment, the software may be stored in memory as an executable program. 
Moreover, parts of the processing of the present invention may be distributed for better 
performance, reliability, and/or cost. 

[0154] It will be readily apparent to those skilled in the art that the invention could be 
practiced with other protocols and/or by encoding, embedding or supplying the data in, 
or along with, the video signal using other encoding, embedding, or marking techniques. 
It may be important, however, that the potentially objectionable material in the program 
be locatable (either temporally, positionally, or spatially) in the audio and/or video signal. 
[0155] The computer program and software modules of the system, method, and 
computer program product of the present invention may be implemented using any 
operating system, and associated hardware including, but not limited to, Unix, Linux, 
VMS, IBM, Microsoft Windows NT, 95, 98, 2000, ME, and XP, Palm OS, Microsoft 
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Windows CE and the like. 

[0156] The systems, processes, and components set forth in the present description 
may be implemented using one or more general purpose computers, microprocessors, 
or the like programmed according to the teachings of the present specification, as will 
be appreciated by those skilled in the relevant art(s). Appropriate software coding can 
readily be prepared by skilled programmers based on the teachings of the present 
disclosure, as will be apparent to those skilled in the relevant art(s). 
[0157] The present invention thus also includes a computer-based product which may 
be hosted on a storage medium and include instructions that can be used to program a 
computer(s) or like device(s) or component(s) to perform a process in accordance with 
the present invention. The storage medium can include, but may not be limited to, any 
type of disk including a floppy disk, optical disk, CDROM, magneto-optical disk, ROMs, 
RAMs, EPROMs, EEPROMs, flash memory, magnetic or optical cards, or any type of 
media suitable for storing electronic instructions, either locally or remotely. 
[0158] The foregoing has described the principles, embodiments, and modes of 
operation of the present invention. However, the invention should not be construed as 
being limited to the particular embodiments described above, as they should be 
regarded as being illustrative and not as restrictive. It should be appreciated that 
variations may be made in those embodiments by those skilled in the art without 
departing from the scope of the present invention. For example, the present invention 
may also be built into a television set as opposed to a stand-alone set top unit. 
While a preferred embodiment of the present invention has been described above, it 
should be understood that it has been presented by way of example only, and not 
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limitation. Thus, the breadth and scope of the present invention should not be limited by 
the above described exemplary embodiment. 

[0159] Obviously, numerous modifications and variations of the present invention are 
possible in light of the above teachings. It is therefore to be understood that within the 
scope of the appended claims, the invention may be practiced otherwise than as 
specifically described herein. 
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